An Algorithm for Estimating the Parameters of Unrestricted Hidden Stochastic Context-Free Grammars

I N T R O D U C T I O N This paper describes an iterative method for estimat ing the parameters of a hidden stochastic contextfree grammar (SCFG). The "hidden" aspect arises from the fact that ~ome information is not available when the grammar is trained. When a parsed corpus is used for training, production probabilities can be estimated by counting the number of times each production is used in the parsed corpus. In the case of a hidden SCFG, the characteristic grammar is defined but the parse trees associated with the training corpus are not available. To proceed in this circumstance, some initim prohabilitie~ are assigned which are iteratively reestimated from their current values, and the training corpus. They are adjusted to (locally) maximize the likelihood of generating the training corpus. The EM algorithm (Dempster, 1977) embodies the approach just mentioned; the new algorithm can be viewed as its application to arbitrary SCFG's. The use of unparsed training corpora is desirable because changes in the grammar rules could conceivably require manually reparsing the training corpus several times during grammar development. Stochastic grammarsenable ambiguity resolution to performed on the rational basis of niost likely interpretation. They also acconnnodate the development of more robust grammars having high coverage where the attendant ambiguity is generally higher. Previous approaches to the problem of estimating hidden SCFG's include parsing schemes ill which MI derivations of all sentences in the t ra ining corpus are enumerated (Fujisaki et al., 1989; Chitrao & Grishman, 1990)). An efficient alternative is the Inside/Outside (I/O) algorithm (Baker, 1979) which like the new algorithm, is limited to cubic complexity in both the number of nonterminais and length of a ~entence. The I /O algorithm requires that tile grammar be in Chonmky normal form (CNF). Tile new algorithm hal the same complexity, but does not have this restriction, dispensing with the need to transform to and from GNF. T E R M I N O L O G Y The training corpus can be conwmiently segmented into sentences for puposes of training; each sentence compris inga sequence of words. A typical one may consist o fY + 1 words, indexed from O to Y: The lookup function W(y) returns the index k of the vocabulary entry vk matching tile word w~ at position y ill tile sentence. The algorithm uses a extension of the representation and terminology used for "hidden Markov modeis '(hidden stochastic regular grammars) for which the Baum-Welch algorithm (Baum, 1972) is applicable (and which is also called the Forward/Backward (F/B)algo~ rithm). Grammar rules are represented as networks and illustrated graphically, maintaining a correspondence ACRES DE COLING-92, NANIES, 23-28 AO~r 1992 3 8 7 I'ROC. OF COLING-92, NANTES, AUG. 23-28. 1992